Add Boson realtime plugin#6340
Open
helin-mktech wants to merge 7 commits into
Open
Conversation
…ntime
Addresses the review finding that a single WebSocket failure was
terminal. Instead of extending the forked connection machinery, drop it
(~400 lines) and inherit the parent OpenAI plugin's
_main_task/_run_ws/_reconnect loop, xAI-style; Boson-specific behavior
now lives in small overrides and hooks:
- Reconnection follows the server's documented contract (the voice-chat
server keeps no state across connections): the base _reconnect re-sends
the full session config and replays the local chat-context mirror via
conversation.item.create, with _create_session_update_event /
_create_tools_update_event / _create_update_chat_ctx_events overridden
to emit Boson-shaped payloads. Retry count/backoff come from
conn_options, and a session_reconnected event is emitted on resume.
- _run_ws wraps the base loop only to reclassify closes a reconnect
cannot fix as non-retryable: close code 3000 (invalid API key) and
server-announced session ends (session.idle_timeout,
session.max_duration_reached), which must not be resurrected.
- Boson-only server events are handled off the base's
openai_server_event_received hook instead of a forked dispatch;
_main_task fails pending response futures immediately on terminal
failure instead of leaving them to their own timeouts.
- Stop advertising auto_tool_reply_generation: the server treats
conversation.item.create as a pure insert now, so the framework must
send response.create after tool outputs — previously tool calls
completed client-side but no reply audio ever came back.
- Treat conversation.item.added for a known id as an in-place update:
the server merges consecutive same-role turns into one item and
re-emits added with cumulative content, which the base insert rejected
("already exists"), leaving the mirror with stale content.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The _handle_response_created override closed the current generation and recorded _current_response_id before the base handler could check _discarded_event_ids. A response that timed out or was interrupted before the server created it would therefore cut off the streams of a newer in-progress generation, and a subsequent interrupt() would cancel the stale response id instead of the active one. Check the discard set first and let the base handler cancel/discard the stale response without touching the in-progress generation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous fix stopped closing the active generation when a discarded response's response.created arrived, but the base handler still parked a _DiscardedGeneration marker in the generation slot, so every subsequent delta of the legitimate in-progress response was silently dropped — and plain save/restore would only defer the damage, because the stale response's response.done (which follows the cancel almost immediately) would close whatever generation sits in the slot: neither the slot nor _handle_response_done is response-id aware. When a legitimate generation is streaming, restore it after the base discards the stale response AND remember the stale response id; _handle_response_done ignores terminal events for remembered ids instead of closing the active generation or clearing _current_response_id. With no generation in progress (the common serial case) the base discard marker is kept so the stale response's trailing events are still eaten. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… ids The server now stores client-supplied item ids and echoes them on conversation.item.added (duplicate ids are rejected instead of merged), so item ids work as the foreign key the base machinery expects. Drop the append-only update_chat_ctx and its manual seen-id ledger in favor of the base implementation: diff against the remote mirror, create/delete sync, and echo-correlated futures. _create_update_chat_ctx_events shrinks to a thin adapter: it feeds the base diff a text-only mirror of the context (audio is represented by its transcript; the GA converter must never see audio frames, whose conversion can raise on empty buffers) and rebuilds each create with the Boson item shape, mapping the unsupported "root" previous_item_id sentinel to a tail append. A minimal extra="allow" item model keeps the base isinstance/echo plumbing working without fighting the GA content-type literals. With items addressable, mutable_chat_context returns to the base default (True): interrupted never-played messages are now cleaned up server-side and realtime sessions can be reused across agent handoffs. Also trim nonstandard surface while here: drop the RealtimeModel.aclose override that closed sessions (session lifecycle belongs to the framework) and the _pending_response_futures test-compat alias, remove the unused NUM_CHANNELS re-export and single-use helpers, warn once instead of per-frame in push_video, and reword comments to describe wire behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A response id enters _boson_discarded_response_ids when its stale response.created arrives while another generation is streaming, and leaves it on the matching response.done. If the connection drops in between, that done never arrives, so the entry would outlive its connection. Clear the set on session_reconnected alongside the other per-connection state, matching how the base reconnect clears _discarded_event_ids. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Boson realtime plugin for LiveKit Agents.
livekit-plugins-bosonwithboson.realtime.RealtimeModel.Notes
["audio"]by default, or["text"]for text-only responses.turn_detection=None.input_audio_transcriptioncontains a non-emptymodel.